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Abstract 

Cognitive structures for expert and novice developers were 
identified through thirteen concepts representing the instructional 
development process. Structures for both groups were analyzed 
according to median (to control for skewed distributions) distances 
within and between concepts describing the Analysis, Design and 
Evaluation components of a learning system design model (Davis, 
Alexander, & Yelon, 1974). For each analysis, results were 
interpreted for distance and dispersion of experts' and novices' 
concept organization by determining the mean (generally considered 
the most stable measure of central tendency) of the medians and 
standard deviations. Consistent with theory, (Geeslin & Shavelson, 
1975; Chi, Feltovich, & Glaser, 1981) findings showed experts' 
organization of concepts within the three components of the model to 
be more consistent with accepted structure of the instructional 
development process; Further analyses of concept organization among 
the Analysis, Design and Evaluation components showed novices' 
structures to be more linear and less integrative in reference to 
the model* Outcomes of this study provide a framework to 

theorstically-intGrpret-ccnstructs-which under lie-successful 

performance in instructional development. Such information can be 
used to guide selection, training and assessment research of 
instructional developers. 
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Past efforts to assess instructional developers have indicated 
that an empirical framework of the instructional development process 
is needed (Faris, 1981; Maxwell & Seyfer, 1984; Bratton, 1984). 
Within instructional analysis approaches, both correlational and 
causal assumptions have been hypothesized regarding the relationship 
between cognitive structure and behavior (Rae, 1986). First, 
knowledge of individuals' cognitive structures may make it possible 
to predict behavior. Second, it may be possible to modify these 
structures (and thus, the behavior itself) by some form of 
training. However, these hypotheses have not been systematically 
tested. 

Procedures used in effective instructional development involve 
a synthesis of information relevant to the task with a repertoire of 
proficient problem-solving strategies (Wagner, 1986). Likewise, 
Jeffries, Turner, Poison, & Atwood (1981) have suggested that 
skilled developers (1) have knowledge about the global structure of 
good development and (2) plan their performance according to 
structural constraints within the learning system. From an 
information processing view, it would be expected that an 

indivi dual Is organization of instructional development_concept«;_in 

memory influences how one perceives, attends to, processes and 
ultimately, solves tasks within the discipline. 

In contrast to behaviorist or associationist theories of 
learning, the cognitive approach postulates that relevant aspects of 
a pre-existing memory structure interact with new knowledge to 
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result in active, meaningful learning. Students who acquire 
concepts early in instruction consolidate more of the learning and 
achieve greater success in problem-solving (Shavelson, 1973). In 
fact, 1t has been suggested that the most significant independent 
factor influencing capacity for acquisition of additional knowledge 
in a discipline is formation of a clear, stabTe, and well organized 
cognitive structure (Ausubel, Novak, & Hanesian, 1968). Shavelson 
(1972) also found that, as students gain new knowledge, their 
cognitive structures begin to resemble relationship hierarchies 
similar to experts. Through identification of individuals' 
cognitive maps, organization of conceptual knowledge may be better 
understood in terms of how these networks represent level of 
expertise. 

Within the past twenty years, research in mathematics and 
science has provided information about the influence of cognitive 
structure organization and procedural knowledge strategies on 
problem-solving performance (Ausubel, et al., 1968; Wittrock, 1974; 
Geeslin & Shavelson, 1975; Greeno, 1978; Gorodetsky & Hoz, 1980; 
Chi, Feltovich, & Glaser, 1981). These studies have shown that 

know.ledgei.structures_of_experts-and_nov-ices-differ— in-domain 

specific organization and that these differences enable experts to 
solve such problems more effectively. Since instructional 
development is essentially applied cognition in problem-solving 
(Wagner, 1986), there is a need for research that can illuminate how 
cognitive structures between experts and novices are organized. 
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This information can be used to further examine knowledge 
compilation (procedural performance) and ways in which developers 
represent and transfer knowledge, and evaluate tasks when solving 
development problems. 

Ultimately, it is knowledge organization and procedures 
activated from this store that determine the quality of problem- 
solving (Gagne, 1985). Problem-solving is often the most common way 
of testing whether individuals meaningfully comprehend the ideas 
they verbalize. However, present assessment techniques to determine 
competent developer performance disregard knowledge investigation in 
favor of performance analysis which lacks empirical validation. 
This course of action is somewhat premature and certainly incapable 
of rational defense. Before measurement of performance can be 
accurately conducted, it is necessary to systematically identify the 
theoretical constructs that underlie successful performance in 
instructional development. 

Therefore, the purposes of this research were to 1) identify 
cognitive networks of concepts within components of a system design 
model (Davis, Alexander, & Yelon, 1974) for expert and novice 

instructional-developer-s,_and-2)-determine-cognitive-networks-of 

concepts between components of the model for both groups in order to 
examine the structure of concept interactions for the instructional 
development process. 

Based on information-processing and assimilation theories, 
(Anderson & Bower, 1973; Ausubel, et al., 1968), inquiry was 
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directed according to the philosophy that relevant, pre-existing 
concepts in memory vary in organization between expert and novice 
instructional developers. Given knowledge of these structures, it 
is. possible to assess how an individual represents and, in turn, 
operational izes the instructional development process. First, 
variables that represent the development process were specified 
according to a learning system design model (Davis, Alexander, & 
Yelon, 1974). This model was used because it has been accepted in 
practice, as one way to represent basic steps of the instructional 
development process. Since there is a vast array of information 
relevant to the discipline, only the most general, inclusive 
concepts were used to provide anchorage in a wide variety of 
instructional development situations. Next, it was determined how 
these concepts are related in the cognitive structures of expert and 
novice instructional developers. Although individuals' networks are 
unique, earlier research has found experts' knowledge organization 
to be more consistent with the accepted structure of a subject 
matter (Gee si in & Shavelson, 1975; Chi, Feltovich, & Glaser, 1981). 
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Materials 

I';-. .' Thirteen concepts were selected according to the system design 
model (Davis, Alexander, !. Ye Ion, 1974) to represent the development 
process: 

Analysis Component 

Performance Objectives (PO) 

LeamejyTrai nee Characteristics (LTCH) 

Task Analysis (TA) 

Needs Analysis (NA) 

Learner/Trainee Prerequisites (LTP) 

Learning/Training Environment (LTE) 
Design Component 

Instructional/Training Strategies (ITS) 

Learning/Training Course (LTCO) 

Course Design (CD) 
Evaluation Component 

Pilot Study (PS) 

Learning/Training Outcomes (LTO) 

Evaluation (E) 

— -Course-Revision— (CR-) 

Three random arrangements of concepts were prepared to control 
for possible sequence effects. 
Subjects 

Ten Individuals (5 experts, 5 novices) in the Educational 
Technology Program at the University of Northern Colorado were 
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chosen on the basis of formal instruction within the discipline. 
The expert group included three faculty members and two doctoral 
candidates (3 males, 2 females). The novice group included four 
doctoral students and one master's student (3 males, 2 females). 
Procedure 

A graphic data analysis procedure (Preece, 1976) was used to 
determine subjects' cognitive structures for the instructional 
development process. Individuals were randomly given one of the 
three equivalent concept lists with instructions. Prior to the 
mapping exercise, subjects were verbally screened to insure that 
they were familiar with the concepts. The mapping activity required 
subjects to look through the list of concepts and initially choose 
the two concepts they believed to be most closely connected, 
individuals crossed these concepts off the list, wrote them down on 
a blank sheet of paper, and connected them with a line labeled as 
"#1". Subjects were then required to examine the remaining concepts 
and choose another concept which they perceived to be the next most 
related to one of the concepts already chosen, or in some instances, 
to another concept on the list. In either case, the same procedure 

was followed to connect concepts. Individua ls continued in this 

manner until all the concepts were joined to at least one other 
concept from the list. Since subjects were individually tested, 
they were not permitted to begin the activity until the examiner 
determined that they properly understood the instructions. 
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Results 

v;^;/.;;v.Tigure;J '.gives' the system design model according to the 
.thirteen concepts used to describe the process (based on Davis, 
Alexander, Yelon, 1974, p. 19). 

^Cognitive maps were scored by counting the number of 
connections (distance) between concepts. Median distances (to 
control for skewed distributions) were computed for all pairwise 
combinations of concepts within and between the Analysis, Design, 
and Evaluation components of the model for both groups. Then, the 
mean (generally considered the most stable measure of central 
tendency) of the medians for each group was calculated to represent 
the average distance of the within and between concepts. Standard 
deviations of the means were used to gauge variability of 
individuals* structures within their respective group. 

The first concern of this study was to identify conceptual 
networks of the instructional development process within the 
Analysis, Design, and Evaluation components of the system design 
model for both experts and novices. Table 2 shows that the mean 
distance and dispersion of scores within the Analysis concepts was 
2.47 (SD « .92) for experts and 2.73 (SD = .96) for novi ces. The _ 
smaller distance for the expert group suggests a more integrated 
cognitive structure. Table 3 shows distances and variability of 
scores within the Design concepts for both groups. Again, experts' 
mean distance and lower standard deviation (2.00, SD » 1.00), 
compared to novices' (2.67, SD = 1.15), signifies that the expert 
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group similarly organized Design concepts to be more compatible with 
the structure of the model shown in Figure 1. Table 4 shows that 
while experts, once more, showed less distance (2.17) than novices 
(2.33), the spread of scores within each group was quite different. 
Experts' lower standard deviation (.75) indicates that their 
structures for Evaluation concepts were more consistent. On the 
other hand, the novice group showed a much higher variability 
(SD « 1.37) and thus, less agreement in their organization of 
Evaluation concepts. 

The second concern of this study was to determine cognitive 
structures for concepts between components of the model in order to 
examine the structure of concept interactions throughout the 
instructional development process for both groups. The same 
procedure was used to determine medians, the mean distance of the 
medians and spread of scores. These analyses revealed that 
different patterns emerged when concepts between components were 
compared. 

Table 5 shows that the mean distance between Analysis and 
Design concepts for experts' (3.50) was greater than novices 
(2.39). However, individual differences within groups showed little 
variability: experts (SD • .86) vs. novices (SD = .92). Table 6 
reveals that experts' structure and spread of scores (3.33, 
SD = 1.37) was more dispersed than novices' (3.08, SD ■ 1.08) for 
Design and Evaluation concepts. And, results given in Table 7 
indicate that organization of Analysis and Evaluation concepts is 
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similar for experts (4.46) and novices (4.58). However, experts 
were less consistent with each other (SD = 1.56) than were novices 
(SD = 1.14). 

Although cognitive structure research is primarily descriptive 
in nature, the means and standard deviations generated from this 
descriptive study provided a basis for statistical inferences. As 
expected, preliminary T-tests for differences in means and F-tests 
among variances suggested that the small sample size (N = 10) did 
not permit a level of power necessary to detect significant 
differences. Further analyses were conducted to explore this 
speculation. When the sample size was estimated at N = 20, results 
showed differences to be significant at the .05 level. 
Discussion 

Concept distances, both within and between components of a 
learning system design model, were computed according to median 
values for pairwise combinations. Average distances for each 
analysis were determined using the mean of the median for each 
group. And, standard deviations were used to examine the spread of 
scores. 

In the first part of the study, results showed that within each 
component, experts demonstrated a more succinct conceptual network. 
These findings are in accordance with the theoretical position that 
experts' knowledge organization is more consistent with the accepted 
structure of a discipline (Geeslin & Shavelson, 1975; Chi, 
Feltovich, & Glaser, 1981). 



12 



•' ; Structure 

12 

In the second concern of the study, concept distances between 
components of the model appeared to be consistent with accepted 
principles of Instructional development. Results coincided with the 
^systems approach" philosophy which postulates that in all phases of 
the development process, the developer must acknowledge that system 
components are interdependent and dynamically Influence each other 
(Davis, Alexander, & Yelon, 1974). This implies a non-linear 
process in which elements within the system interact throughout the 
entire course of development. Thus, it should be expected that the 
expert developers would consider Interactions among concepts and 
consequently display greater distances in their structures for 
Analysis-Design and Design-Evaluation comparisons* Conversely, \% 
should apply that a novice approach towards development is more 
linear and therefore, distances between concepts would be smaller in 
Analysis-Design and Design-Evaluation comparisons. Results of the 
final contrast, Analysis-Evaluation, were also explainable. 
According to heuristics of instructional development (Davis, 
Alexander, & Yelon, 1974), planning for evaluation should originate 
in the Analysis stage. Thus, it would be expected that concept 
distances for both groups should be somewhat similar. 

The findings of this descriptive study provide information 
needed to empirically identify and interpret the theoretical 
constructs that underlie successful performance 1n instructional 
development. These preliminary results suggest that experts and 
novices differ in their conceptual organization, both within and 
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between components of Instructional development process. According 
to theory (Chi, Feltovich, & Glaser, 1981), these differences should 
enable experts to solve instructional development tasks more 
effectively. 

~ The study should be replicated with a larger number and more 
diverse group of instructional developers to further define 
distinctions found in this research and better facilitate 
inferential analyses. Future studies also need to investigate 
(1) If there are significant differences in concept organization 
among individuals within their respective groups (2) how the 
direction of concept organization in cognitive structure relates to 
accepted instructional development procedures, and (3) how 
conceptual organization of trie instructional development process 
relates to performance. Answers to such questions can provide a 
link between theory and practice. Educators and practitioners need 
this integration to devise rational, defensible, and empirically- 
validated procedures for selection, training and assessment of 
instructional developers. 
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